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The  ne search  study,  Virtualization  JS’/icircJsr  Feosptiin(y  and  Implementation  in  tl le 
USNA  Computer  Science  department  was  conducted  at  the  United  States  Naval  Academy 
in  an  eFFbn  to  help  define  a  how  sharing  virtual  machines  which  had  been  transferred  via 
external  hand  drive  From  host  Lo  host,  and  run  on  VMware  workstation,  could  be  run  on  a 
single  powerFul  server  and  require  users  to  interact  with  them  using  a  thin  client.  Specific 
topics  cover  basic  virtualization  concepts,  dillerences  in  architecture  between  Xen  and 
VMware,  and  the  performance  seen  on  a  test  network  utilizing  one  server  running  ESX, 
As  corporations  and  other  large  entequrises,  including  the  Department  oF  DeFense, 
move  from  the  traditional  physical  server  infrastructure  towards  virtual  consolidation, 
study  in  this  area  becomes  more  and  more  pertinent.  In  the  USNA  Computer  Science 
□e panmen t,  this  server  resides  on  a  sandboxed  network,  used  only  For  testing  purposes, 
but  this  technique  has  been  implemented  across  many  major  organizations  running 
servers  as  a  result  of  low  utilization  of  traditional  physical  infrastructure.  Using  a 
virtu  alLzed  architecture  allows  more  dynamic  load  sharing  based  on  rite  curreni  demands 
placed  on  a  particular  host,  and  overall  results  in  less  idle  time  on  the  infrastructure. 

The  goal  of  this  research  paper  was  to  define  potential,  architectures  that  satisfy 
our  existing  needs,  including  labs  for  Information  Assurance  classes,  exercises  such  as 
Cyber  Defense  Exercise,  and  development  work.  By  analyzing  their  relative 
performance,  a  compromise  between  performance,  ease-of-use,  and  the  resources  of  the 
□e  panmen  I  provided  recommendations  thar  will  become  an  integral  pan  of  Computer 
Science  and  Information  Technology  education. 

At  the  conclusion,  numerous  studies  on  both  VMware  and  Xen  architecture  were 
analyzed,  which  gave  insight  into  architectures  to  be  modeled  by  the  Department,  For  the 
purposes  oF  research,  Xen  was  Focused  on  more  heavily  by  nature  of  being  open  source. 
However,  our  current  VMware  license  weds  us  to  their  infrastruciure,  l he  main  reason  For 
solely  analyzing  ESX.  This  study  may  also  lead  to  Further  research  into  topic  areas  such 
as  dynamic  image  swapping  across  multiple  servers,  vulnerabilities  oF  virtualization 
shares,  and  even  more  utile  architectures  For  the  Department's  needs. 


L  [ntroilucdoii 


As  tin?  cost  of  producing,  advanced  server  hardware  with  greater-than-necessary 
capability  continues  to  decrease,  virtualization  lias  become  even  more  crucial  and  widely 
utilized  in  server  operations,  Ratlier  than  let  this  power  go  unused,  virtual  server 
environments  seek  ro  provide  many  environments  on  a  single  or  small  number  of  servers, 
with  each  vinual  server  serving  a  distinct  purpose.  In  the  case  of  the  Naval  Academy's 
computer  science  department,  these  hosts  function  as  experimental  machines  for 
Information  Assurance  classes,  which  act  on  both  the  offensive  and  defensive  sides  of  dll 
Injections,  port  scanning,  and  other  basic  hacking  techniques.  My  goal  in  this  paper  is  to 
examine  the  feasibility  of  establishing  a  shared  server  for  these  images,  so  diat  rather  than 
copying  a  prepared  image  to  each  individual  host  either  through  FTP  or  various  hard 
drives,  they  could  be  modified  on  a  single  hosL,  uploaded  to  the  server,  and  have  groups 
of  Midshipmen  modify  them  simultaneously.  In  the  worst  case,  the  share  would  be 
configured  as  usual,  but  copies  of  images  would  be  "pushed”  to  or  “pulled”  from  hosts 
running  a  compatible  client, 

2.  I.iui  a i hi  e  Review 

There  are  two  types  of  vinualizationr  process  and  system,  “Process 
virtualization”  is  barely  regarded  as  such  in  the  sense  Lite  tenm  is  used  today,  as  it  has 
become  a  requirement  in  kernel  operations,  and  goes  unnoticed  in  everyday  computing 
applications.  In  process  virtualization,  each  process  is  allocated  its  own  address  space, 
registers,  and  file  structure.  Albeit  small,  tliis  constitutes  a  virtual  environment, 
especially  when  a  critical  system  process  is  permanently  assigned  these  attributes,  much 
like  a  chnoot  jail  assigns  a  process  a  smaller  filesystem,  “System  virtual ization/1  the 
primary  focus  of  this  paper,  is  when  a  relatively  powerful  host  aflows  guet  operating 
systems  to  run  within  the  existing  operating  system  (Smith  and  Nair,  2005),  On  any 
given  machine,  a  cenain  Instruction  set  architecture  (ISA)  is  utilized  by  a  user  through 
Ihe  operating  system  to  achieve  desired  effects  from  the  machine's  hardware.  Because 
Microsoft  Windows'  ISA  does  not  mesh  nicely  with,  that  of  a  Debian  Linux  variant's, 
virtualization  software  such  as  VMware  serves  as  an  intermediate  to  allow  a  vinual 
image  of  one  operating  system  to  run  inside  the  other,  This  software  allocates  memory, 
processor  time,  and  bandwidth  among  other  limited  performance  factors  from  the  host 
itself  to  the  vinual  machine(s)  running  within.  Examples  of  this  software  fora  single 
host  include  VMware1  Workstation,  and  Sun'  Virtual  Box, 

One  of  the  prime  experimental  mediums  for  l his  study  will  be  VMware  ESXi 
server,  ESXi  and  ESX  are  known  as  Type-1  hypervisors,  which  means  they  run  directly 
on  a  powerful  host's  hardware  and  act  as  a  monitor  and  a  share  for  the  vinual  hosts  that 
run  on  them,  Both  sers  of  software  contain  what  will  be  referred  ro  throughout  the  course 
of  this  study  as  vinual  machine  monitors  (VMM's),  tocomrol  various  parameters  crucial 
to  virtualization.  On  any  virtualization  server,  the  hard  disks  of  the  host  are  partitioned 
into  separate  shares  for  each  hosted  VM  (Vinual  Machine).1  For  the  puqjoses  of  this 
study,  this  powerful  host  will  be  a  Dell  PowerEdge  2950  Server'  running  VMware  ESXi, 
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Another  Type-1  hypervisor  is  Cirri';1  XenServer,  which  has  a  much  lower 
fingerprint  on  the  server  it  runs  on  than  VMware  H;  ESX.  In  an  ESX  environment,  the 
guest  host  is  “unaware'1  that  it  is  being  run  virtually,  which  requites  proprietary  V  M ware 
drivers  loreach  server  component  that  the  VM's  require  access  to  (i,e.  Network 
interfaces,  storage  configurations),  XenServer,  on  the  other  hand,  makes  it  obvious  to 
the  guest  03  rhat  it  is  being  run  virtually  find  interfaces  with  the  Xen1  management 
console,  known  as  “Domain  O,'1  which  then  interacts  with  the  hypervisor  through  open 
source  drivers,  “Domain  0,J  is  actually  a  separate  VM  running  a  hardened  and  optimized 
Linux  kernel.  This  process  is  known  as  para^rfLafizotfon,  which  works  almost 
seamlessly  for  Linux  OS's  on  both  the  VMware  and  Xen  hypervisors,  Windows 
operating  systems,  by  contrast,  cannor  be  fully  para  virtual  Lzed,  and  for  certain 
instructions  rely  on  the  host  hardware's  virtualization  assist  technologies.' 

The  primary  areas  this  study  tv  ill  Joclis  on  that  we  expect  to  limit  our  setup's 
capabilities  are  disk  input- output  (I/O)  rates,  virtual  memory  allocation,  viitual  network 
interface  card  (NIC)  availability,  and  cennal  processing  unit  (CPU)  sharing,  in  increasing 
ordeL  of  effect.  Many  methods  of  improving  virtual  ization  sharing  have  been  put  forth  in 
various  studies. 

One  such  method  is  VMM  I/O  bypass,  in  which  interactions  with  devices 
themselves  go  through  a  separate,  specially  designed  VM,  vice  the  VMM.  By 
eliminating  the  VMM  in  this  operation,  an  area  of  bottlenecking  is  eliminated,  and  may 
help  the  spread  the  load  across  different  platforms.  Unfortunately,  while  this  is  possible 
using  the  Xen  architecture,  which  is  based  heavily  on  para  virtual  Ization,  ESX  Ltquires 
that  all  I/O  operations  go  through  the  VMM  (Liu  2006),  As  seen  in  Frbning's  study,  this 
serves  as  a  limiting  factor  in  performance,  as  V NT's  require  extensive  I/O  operations 
(Froning  2009}. 

A  proposed  way  to  drastically  reduce  Lliis  overhead  is  through  concurrent,  direct 
network  access  (CDNA),  As  compared  to  the  traditional  infrastructure,  in  which  the  each 
VM  interacts  with  a  separate  back-end  driver  in  the-  hypervisor,  CDNA  utilizes  a  single 
CDNA  NIC,  which  all  the  VM's  interacr  with  and  rhe  hypervisor  still  monitors.  Versus 
traditional  Xen  and  VMware  implementations,  Lhis  yields  performance  improvements  of 
371QFX)  transmit  throughput  and  12636  receive  throughput  in  a  24  VM  semp.  As  of  2006. 
this  has  yet  to  be  implemented  in  either  of  Lhe  two  vendors'  products  (Risnei.  2006). 

This  network  and  I/O  overhead  is  a  key  concern  when  considering  whether 
purchasing  an  ESX  infrastructure  will  lot?  worth  its  cost  to  the  Department,  A  June  2009 
study  on  parallel  computing  using  VM's  on  an  ESX  in  least  ructULe  found  that  the  I/O 
limitations  of  the  setup  yielded  significantly  slower  i untimes  when  more  than  32  VM 's 
were  utilized  simultaneously  (Martinez  2009).  This  was  mainly  due  to  inter¬ 
communication  overhead  both  on  the  VNI's  and  in  tine  hypervisor.  This  setup,  however, 
used  processes  that  were  continually  communicating  and  executing,  For  the  purposes  of 
the  department,  our  Traffic  and  infrastructure  utilization  will  come  in  spurts  as  computers 
aie  scanned,  at  Lacks  are  executed,  and  defenses  are  put  in  place,  so  our  performance 
should  be  significantly  better, 

J.  Tesl  and  Analysis 
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This  study  seeks  to  find  an  efficient  way  of  creating  a  virtual  setup,  usually  of  one 
to  three  virtual  machines  on  a  bridged  adapter,  modified  to  fit  the  purposes  of  an 
Information  Assurance  training  lab,  and  propagating  that  setup  to  a  clusrerof  hosts, 

None  of  Ihe  hosts  will  be  paravi dualized  to  ensure  reliable  test  results  across  virtual 
platforms.  If  necessary,  the  setup  could  be  “pull”  iDased.  meaning  the  liost(s)  would 
connect  to  the  server  and  request  the  set  of  images  through  VMWare  software.  If  this  is 
infeasible  due  to  licensing  cosrs  for  additional  software,  or  due  ro  bandwidth  restrictions, 
the  server  could  simply  be  utilized  as  an  FTP  share.  Where  ESX  truly  has  great  potential, 
however,  is  the  simultaneous  modification  of  images  running  on  the  server.  However, 
this  would  require  more  coordination  between  lab  partners,  and  rewriting  most  of  the  Labs 
for  Information  Assurance  and  Advanced  Information  Assurance,  as  they  are  based  on 
single  host  setups. 

In  order  to  test  these  requirements,  this  study  tests  four  major  aspects,  First  i! 
tesrs  the  use  of  linked  clones  running  off  a  Samba  file  share,  followed  by  using  the  server 
as  a  file  share  for  images,  then  moves  into  more  advanced  architecture  provided  by 
vSphere,  Tit  is  entails  simultaneously  modifying  Lhe  images,  and  testing  their 
performance  using  key  metrics  such  as  memory  consumption,  virtual  CPU  strain,  and 
network  bandwidth. 

To  minimize  our  test  environment,  configurations  will  first  be  tested  with  a  single 
host,  then  with  four  hosts  (ref.  Figure  1).  If  a  test  run  shows  a  great  deal  of  promise,  i.t 
can  be  run  again  with  the  full  sixteen  hosts.  The  overall  assumption  for  our  original 
configuration,  with  an  instructor  preparing  images  on  another  workstation  and 
transferring  them  using  a  USB  hand  drive,  is  that  movement  to  each  host  takes  roughly  13 
minutes. 


( 1 3  min  )  /  (2  har d-drivcs)  =  7.5  minutes/  Image  +  overhead  of  moving  around 
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Tesl  Procedure  J ;  Using  tlie  existing  CS  Deportment  license  lor  VMware  workstation, 
linked  clones  were  created  and  run  off  the  “Tera”  server. 
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'Desktop,'  as  rhe  time  to  configure  the  virtual  network  of  the  images  would  have  taken 
additional  time. 


Conclusion:  Without  an  incredibly  expensive  infrastructure  which  allows  for  extremely 
fast  disk  reads  and  writes,  or  a  more  efficient  file  sharing  protocol,  this  setup  is  nol 
feasible. 

Test  Procedure  2:  Virtual  machine  created  on  ESX  host  through  a  vSphere  client, 
transferred  over  FTP  by  cluster  of  hosts  using  the  same  client, 

[He tails:  Clients  simply  browse  the  datastore  (this  would  require  another  user,  for  example 
’iastudent,'  bur  for  purges  of  testing,  'chris,'  a  member  of  the  local  administrators  group 
was  used),  and  this  user  saved  the  entire  'Backtrack  4'  directory  to  'Desktop.'  The  vdmk 
hard  drive  (the  main  source  of  any  delay,  as  configuration  and  log  file  sizes  are 
essentially  negligible)  is  a  flat  5  gigabyte  virtual  disk. 

Findings:  Data  transfer  rate  ceilings  at  roughly  45,000  Kbps  (ref.  Figure  2),  causing  the 
rransfer  to  take  roughly  15  minutes.  On  the  single  host  test,  results  were  comparable,  as 
the  test  took  roughly  4  minutes. 

(  15  win  /  /  (  4  hosts  )  x  ( 16  hosts  )  =  6’d  mimitcVlmoge 
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Figure  2:  Data Transfer  Rales  on  ESX  host 

Conclusion:  While  this  doubles  the  speed  of  the  existing  procedure,  it  still  does  noL  meet 
our  requirements.  The  process  is  still  quite  slow,  even  when  using  a  switch  instead  of  a 
hub.  Additionally,  if  administrator  access  is  required  to  access  these  images,  the 


instructor  would  have  ro  log  them  off  upon  completion,  so  they  couldn't  just  bit  download 
and  leave  the  lab.  This  assessment  also  doesn't  consider  the  case  where  the  size  ol  rite 
hand  drives  is  greater  titan  5  CB;  using  simple  math  it  would  increase  the  time  to  get  the 
images  out  by  f  f  [sfzc  in  GE]  /  [  5  CE  ]  -  l  )  *  100  )  percent,  without  factoring  in 
overhead. 


Test  Procedure  3:  Virtual  Machine  created  on  ESX  Host  through  vSpliere  client,  pushed 
or  pulled  by  clients  at  one  Lime.  This  will  be*  considered  to  be  negligible  in  the  future, 
because  the  transfer  analyzed  in  this  test  will  be  that  of  finked  clones.  Linked  clones  are 
essentially  a  track  record  oi  changes  made  to  a  virtual  machine,  so  over  time,  the  one 
time  cost  of  transferring  the  image  to  the  host  becomes  negligible.1 

Details;  Same  setup  as  above.  The  current  bang-up  with  this  setup  is  that  the  department 
lacks  a  full  license  for  vSphere,  which  limits  a  user  from  creating  a  clone.  Tine  ESX  (Jt 
on  the  host  itself  does  not  include  a  feature  to  pusli  the  images. 

Ejlltliugs:  Since  VMware  workstation  and  ESX  are  not  “our-of-th  e-box”  compatible,  the 
current  method  of  placing  an  existing  image  on  rhe  ESX  server  has  been  to  create  a  fresh 
image  ol  the  same  OS  on  the  server,  and  instead  ol  creating  a  new  virtual  disk,  adding  Lite 
existing  virtual  disk  as  rbe  hard  drive.  This  often  requires  a  simple  hack  io  fool  die  newly 
created  machine  into  believing  tine  hard  drive  is  native  to  it:  naming  everything  exactly  as 
it  was  in  rhe  original  image.  This  lias  been  of  particular  concern  for  Windows  images 
whose  hard  drives  have  been  split  into  2CB  chunks.  This  manner  ol  partitioning  requires 
numerous  extra  configuration  files,  which  must  be  copied  into  rhe  correct  directory  under 
Hie  proper  name  in  order  to  function  properly,  Linux  hosts,  especially  these  with  ‘‘flat" 
virtual  storage  allocations,  are  nor  problematic  at  all.  Currently,  this  seiup  could  be  used 
as  a  laackup  to  other  methods,  but  would  not  be  fully  utilizing  the  expensive  vSpliere 
license. 

Tesi  Procedure  4:  Vinual  Machines  will  be  modified  simultaneously,  and  the  Labs  for 
each  class  or  setup  the  vinual  ization  share  would  be  used  in  would  be  rewritten  to  fit  this 
model. 

Details:  Tine  re  are  roughly  10-12.  Labs  for  each  JT430and  1T432  class.  Teams  would 
consist  of  4  people,  or  however  many  sat  at  one  table,  Sr  ness  testing  for  the  server 
consisted  of  running  6  hosts  on  the  server  f Backtrack,  2- Windows  XP,  Ubuntu  9. 10  beta, 
and  2-Ubunru  9.04  servers},  ping  floods  from  all  Linux  guesrs,  intense  pon  scans  from 
Windows  guests,  and  infinite  'while'  loops  on  all  hosts. 

Findings:  In  Ihe  ideal  case,  while  users  end  up  “fighting”  for  the  keyboard,  it  encourages 
collaboration.  If  one  group  member  is  more  knowledgeable  dun  the  other,  he  can  do 
what  lie  needs  to  to  the  image,  and  say  out  loud  to  the  group  what  he  is  doing,  thereby 
insiructing  everyone  at  rite  table  without  students  having  ro  look  over  one  another's 
shoulder.  Running  them  on  the  server  itself  eliminates  the  “pusli”  or  “pull”  from  the 


4  Pmoeduie  detailed  in  Appendix  A 


clients  3] I  together,  so  time  to  execute  the  lab  is  negligible.  Stress  testing  the  server 
yielded  no  noticeable  changes  to  the  end  user,  but  CPU  usage  on  the  ESX  server  spiked. 
The  only  time  considerations  in  this  setup  are  tine  preparation  time,  and  changing  line  labs 
to  meet  this  new  method  of  practicing  LA,  The  former  lias  never  been  considered  in  this 
analysis,  and  therefore  will  not  count  toward  the  time,  Re-writing  Hie  labs,  however,  is  a 
serious  time  consideration  with  great  potential. 

From  our  testing  of  an  Advanced  Information  Assurance  lab  involving  three 
images,  users  were  not  as  Veen  on  the  collaboration  as  originally  expected.  Likely,  this 
was  a  result  of  precedent,  since  lor  one  and  a  half  semesters  of  Information  Assurance 
classes,  individual  images  had  been  provided  to  users  on  each  host.  The  day  of  the  test, 
and  only  minutes  before  they  started  the  lab,  this  new  approach  of  simultaneous 
modification  was  Introduced  to  the  students,  which  may  have  been  the  reason  for  ilieir 
confusion. 

Where  this  setup  holds  the  greatest  potential  is  a  hacking  competition  against  a 
|Darricular  target  image.  Access  permissions  on  the  Large t  could  be  limited  to  die 
administrator  or  instructor,  so  users  could  not  open  a  console  and  modify  the  image. 
Better  yet,  they  could  be  set  as  view-only,  so  that  the  user  could  see  what  effects  their 
attacks  had  on  the  box,  without  any  ability  to  execute  programs  on  the  image.  If  a  simple 
script  which  ran  lnetstat'  in  a  time  loop  was  used  on  the  target  image  along  with  a 
process  listing,  the  end  user  could  also  view  how  stealthy  any  connections  they  made  to 
the  box  would  be,  and  this  host  could  even  be  projected  up  onto  the  screen  so  all  teams 
could  monitor  their  progress.  In  this  case,  however,  virtual  hardware  availability 
becomes  a  concern.  Consider  RAM: 

(  4  rrams  )  *  [(  2  Backtrack  wnogcs  }  *256  iWB  +  (  1  XP  Image  )  *  512  MB 

+  (  1  Ubmtu  Image  )  *  384  MB  +  J 
+  I  Target  linage  *  512  MB 
=  ~  8144  AJB  of  RAM 

Values  based  upon  VMwarc  recommendations  and  IT430/2  precedent 

fl  192  MB  is  the  available  RAM  for  the  current  serup,  and  since  ESX  idles  around 
800  MB  of  usage,  only  7392  MB  Is  available. 

Virtual  CPU's  an?  another  consideration: 


4  trams  *4  images  +  1  target  image  =17  images  -  17  Virtual  CPU's 
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Figure  3:  CPU  Performance  During  Stress  Testing 

With  the  current  setup,  there  are  only  8  CPU's  available.  Further  dividing  them  to 
fit  thislab  setup  will  decrease  performance  and  gieatly  increase  stiess  on  the  server  (ref. 
Figuie  3).  Even  though  ESX  allows  this,  the  swapping  to  allocate  CPU  time  to  a  given 
host  does  not  allow  a  full  piocessor  to  be  allocated  to  eacli  V M  in  the  case  where  all  4 
teams  aie  using  half  of  their  images,  or  vice  veLsa.  Tit  is  will  be  examined  more 
methodically  later  in  tine  analysis. 

Test  Procedure  5:  Open  Virtualization  Format  (QVF1  implementation.  The  template 
would  be  deployed  to  the  whole  server,  and  should  allow  for  compatibility  between  the 
Linux-based  ESX  server  and  tine  Windows  sandbox  machines  running  VMwaie 
workstation. 

Details^  OVF  provides  moie  data  about  a  machine  than  a  vtlmk  virtual  disk  can,  to 
include  memory  allocations,  CPU  allocations,  display  name,  referenced  configurations 
files,  device  nodes,  and  network  configurations,  it  is  a  complete  specification  of  the 
virtual  machine  or  set  of  viitual  machines.  Fiom  vSphere,  this  is  as  simple  as  a  point  and 
click,  Potential  exisis  in  packaging  our  current  images  or  lab  setup  into  this  format,  so 
that  it  could  simply  but  uploaded  to  the  server  and  run  Lime,  or  run  from  an  active 
administrative  client  with  the  OVF  file  on  tine  local  disk." 


r. 


Imp:  f.'ww  w.youLU  be.cnrnnv3tdi?v=OtflU  VFI  t7m  Efic  Feat  u  if=pl:tyei  _pml3edded 


Findings:  While  supposedly  cross-platform  and  cross-vhtualizBtlon-sofrware  compatible, 
this  is  not  the  case.  The  best  example  was  attempting  to  port  fresh  Virtual  Box  images  to 
VMWare,  or  vice  versa,  Even  without  their  respective  toolkits  installed,  these  OVF 
templates  have  a  host  of  problems:  not  being  able  to  set  configuration  parameteis,  not 
being  able  to  lecognize  the  hard  drive,  even  with  the  proper  file  vdmk  file  extension,  and 
not  being  able  to  recognize  the  OVF  configuration  file  because  of  different  namespaces 
and  the  number  oF  device  nodes  supported.  What  was  helpful  about  VMware  to  VMware 
OVF  transfers  was  that  it  generally  compressed  the  hard  drive,  reducing  File  transfer  time. 
If  an  FTP-based  solution  was  required,  this  would  seive  as  an  excellent  way  of  packaging 
the  images. 

Test  Procedure  6;  Limited  resources  are  an  ever-present  problem  with  virtualization 
software.  Considering  that  our  ultimate  goal  for  this  server  is  to  run  a  large  number  of 
images  to  support  a  class  or  competition,  two  key  areas  of  consideration  come  to  mind: 
RAM  and  CPU  limitations.  This  test  procedure  focuses  on  CPU  overloading,  which  may 
become  problematic  when  each  machine  is  allocated  a  set  number  of  virtual  CPU's,  and 
the  sum  oF  allocated  virtual  CPU's  becomes  greater  than  the  number  of  physical  CPU's 
available. 

In  Older  to  test  the  eFfects  this  would  have  on  our  current  setup,  we  booted  all  the 
virtual  machines  From  our  'Cyber  Weekend'  resouioe  pooL  totaling  12  virtual  machines, 
each  with  a  single  virtual  CPU  allocated.  Our  current  ESX  setup  contains  4  CPU's  with  2 
cotes  each,  for  a  total  of  fi  possible  dedicare d  CPU's,  assuming  simple  lead  sharing 
practices.  With  ESX  acting  properly,  we  would  anticipate  that  load  sharing  and  CPU 
swapping  would  occur.  Upon  boor,  we  had  some  spikes,  up  ro  75%  on  a  single  CPU,  but 
after  settling  into  boot  mode,  they  idled  under  5%, 

To  introduce  extreme  loads  on  the  CPU,  we  put  every  virtual  machine  in  an 
infinite  loop: 


Wi  ii  1.1'  it\  ft : 

Ij  i  ri  1  i  ri  i  ..h  I  j:i  .  :]-±L 

MW  ECHO  OFF 
=  1 

echo  1 rf 

qoL.o  1  : 


T.irrux: 

I  .  / b  i  i  / 1  id  ^  hi 

while  tree;  c.c- 
echo  "test",- 
dor.e 


crv. -RcjRhK-..  jit >,-'1009  1(1:30:31!  -  ii.-i/:2D0D  il:3«:2p  AM  -  incaiwt.lxalsiDflialn 


Figure  4:  CPU  Performance  during  Inflniie  Luup  Stress  Testing 

Although  it  functioned  properly  in  our  stress  testing,  tills  is  obviously  not  a 
sustainable  mode  of  operation  without  a  much  more  reliable  cooling  mechanism  for  the 
server  [ref.  Figure  4),  Our  earlier  test  setup,  where  CPU  usage  gradually  ramps  up,  is  a 
much  more  realistic  scenario. 

The  /act  that  two  VM’s  were  booted  while  all  system  CPU's  were  running  at  close 
to  capacity  leads  me  to  conclude  dial  that  the  load  sharing  in  ESX  server  may  allow  the 
server  to  temporarily  overcome  limited  resources,  but  it  is  nol  desirable  when  any  sort  of 
performance  is  required. 

Test  Procedure  7:  In  order  to  see  the  effects  of  over-committing  our  server's  memory 
resources,  we  ran  a  sim  ilar  setup  to  the  CPU  test,  we  ran  4  guests  at  2GB  of  dedicated 
memory  each,,  which  should  take  our  memory  over  the  edge  and  generate  memory- 
sharing  or  extensive  paging,  Each  VM  was  allotted  a  single  virtual  CPU,  as  in  our  last 
setup,  so  as  noL  to  make  virtual  processors  a  limiting  factor, 

4  VutLfcif  Machines  x  [  (  2048 MB  allocated  memory )  +  ~  100  MB  Overhead ) 

=  -  8592  MB  Allocated 

£1592  MB  '■  6192  MB  physical  server  memory 

When  we  added  a  test  machine,  a  Windows  XP  with  256MB  of  HAM,  with  4 
others  allocated  above  the  memory  threshold,  there  was  no  effect  on  the  machines 


execution,  which  appeared  to  he  a  result  of  ESX  independently  adjusting  our  allocations. 
Allocating  5192  MB  of  RAM,  the  "gran ted”  memory'  for  the  system  idles  at  7,500.000 
Kin,  or  7324  ME.  To  ensure  we  went  over  the  threshold,  we  then  started  booting 
machines. 

This  has  no  effect  on  ESX.  In  order  to  ensure  the  accuracy  of  our  findings,  1 
coni  in  ued  to  boor  virtual  machines,  and  ran  memory  Intensive  processes  on  all  of  them,  to 
no  avail,  line  only  time  an  impact  was  observed  was  when  1  ran  simple  programs  on 
guests  to  infinitely  allocate  memory  in  chunks: 

.i(Ed_meinory-c: 

It  include  ■Catd.lib  .  h> 

int  ma i n  (  )  | 

while  ( 1 ) 

mal 1 oc ( 1 024 ] ; 
return  0 ; 

} 

This  affected  both  the  servers  performance  monitor  and  that  of  tine  VM  itself. 
Otherwise,  Irom  the  server's  performance  monitor,  it  simply  showed  that  the  server  had 
allocated  ail  its  available  memory  (■-'7500MB.  with  overhead  =  5192  MB). 

This  result  is  validated  by  Waldspurger's  study  as  Vmware  has  implemented  since 
early  versions  of  ESX  the  ability  for  memory  to  “balloon'1  Ixised  on  the  immediate 
requirements  by  the  VM's  at  any  given  time,  Many  algorithms  have  been  developed  to 
efficiently  manage  this  sharing  by  VMware  and  independent  scholars  alike,  which  is  one 
of  the  reasons  forESX's  wide  deployment.  Another  is  that  this  feature  distinguishes  it 
from  Vmware  Workstation,  which  lias  a  policy  of  not  being  able  ro  over-all ocate  memory 
without  informing  the  user;  the  dialog  in  Workstation  reads  “memory  swapping  may 
occur,  reducing  performance,'1  With  ESX.  memory,  CPU,  and  resource  swapping  in 
general  is  considered  implicit,  hence  its  built-in  design  to  manage  such  loads 
(Waidsbunger  2009), 

Test  Procedm e  it:  With  CPU  and  RAM  being  efficiently  managed  by  ESX.  another 
remaining  factor  was  how  network  traffic  affected  the  host, 

Details:  5  VM's,  ail  Linux  based,  were  booted  and  each  allocated  a  single  Virtual 
CPU  and  512  ME  of  RAM.  Later,  the  number  was  expanded  lo  5,  but  ro  ensure  Virtual 
CPU's  were  not  a  limiting  factor,  6  were  used  for  our  initial  test,  IP's  were  statically  set 
in  increments  of  10.  and  each  host  ran  a  ping  flood  on  the  subsequent  bast  in  the  chain 
(i.e,  10, 10,10,20  pinged  10.10.10.30,  while  being  pinged  by  10.10.10.10).  This  generated 
network  bandwidths  of  200-300  KB/s,  where  the  host  did  not  Jim  it  the  number  of 
outgoing  packets.  This  test  yielded  the  most  curious  results  of  all:  on  the  process  monitor 
within  the  VM,  little  to  no  CPU  activity  was  observed.  On  the  server's  performance 
monitor,  however,  the  CPU's  appeared  to  be  maxed  out,  and  even  over-maxed  on  those 
hosts  that  were  running  an  unlimited  ping  flood. 

Findings:  VM  performance  slowed  on  some  machines,  especially  once  they  began 
both  sending  and  receiving  packets.  When  8  bests  were  mu  in  the  same  ping  loop,  the 
problem  became  worse.  While  no  official  “benchmark'1  was  taken,  the  test  i  ran  started 


with  simple  operations,  such  as  opening  a  terminal,  then  moved  on  to  more  complex  task, 
like  opening  a  web  browser,  then  the  openofflce.org  suite.  The  delays  experienced 
appeared  to  be  related  to  X  rendering;  when  a  terminal  was  opened,  first  a  white  box 
would  appear,  the  colors  would  render,  rhen  a  full  shell  would  appear.  This,  on  average, 
took  3-5  seconds.  Normal  operation  is  almost  instant.  Typing  delays  occurred,  as  well,, 
which  hovered  between  0.5  and  J  .0  seconds  just  to  type  into  the  web  browser  and  the 
terminal.  These  tests  were  run  on  Linux  with  a  system  monitor  up,  and  the  CPU 
experienced  no  spikes,  ESX's  view  of  the  operation  was  much  different; 
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This  likely  spawns  from  the  setup's  Lack  oi  physical  NIC’s  for  each  VM,  which  is 
a  common  problem  to  any  virtual  architecture.  As  evidenced  by  the  guests'  system 
monitor  view  of  tire  CPU  time,  the  ping  flood  itself  is  not  taxing  to  the  CPU,  However, 
tine  physical  CPU's,  which  are  assigned  as  Virtual  CPU's,  have  to  process  all  incoming 
and  outgoing  network  packets.  In  the  scenario  we  demonstrated,  this  was  especially 
5t renuous  because  all  the  guests  were  both  sending  and  receiving,  and  in  the  case  of 
Ubuntn,  rendering  a  robust  X  window,  which  on  a  physical  machine  would  be  handled  by 
a  separate  GPU, 

Framing's  st tidy  backs  this  hypothesis.  Virtual  network  interlaces  [VN1),  by 
design,  require  more  work  with  respect  to  the  virtual  CPU  assigned  to  the  machine,  since 
tine  same  CPU  needs  to  analyze  which  processes  are  sending  data  across  the  network. 
More  specifically,  it  needs  to  package  the  data,  and  schedule  when  these  packets  can  go 
across  the  network,  VNTs,  Like  any  network  interface  card,  require  a  queue,  but  the  fact 
that  a  vinual  PID  lias  to  be  inserted  into  this  queue  in  addition  to  the  data  itself  means 
that  it  requires  a  read  to  determine  whether  space  exists  on  tine  queue,  and  a  write  to 


actually  insen  the  data.  Combined  with  overhead,  all  these  facrors  make  rlie  VNi 
significantly  less  efficient  than  a  physical  NIC,  and  account  for  Lhe  extreme  stress  on  Lbe 
CPU's  when  6  to  S  VNl's  are  sending  and  receiving  simultaneously  (Froning  2009}, 


4  Summary  and  Cmidusion 

Based  on  the  above  tests,  the  most  problematic  area  of  the  network  is  virtual  CPU 
availability,  since  it  play's  a  role  in  so  many  operations  that  are  otherwise  handled  by 
physical  hardware.  This  is  especially  evident  with  the  VNf,  a  crucial  part  of  the  setup 
considering  that  extensive  traffic  will  be  passed  across  the  network  to  scan,  exploit,  and 
deny  service  to  hosts  by  many  users  simultaneously,  While  the  load  will  rarely  be  as 
focused  as  the  one  experienced  during  stress  testing,  more  users  than  the  number  used  for 
testing  will  be  using  the  images  on  the  server,  many  of  which  win  be  doing  RAM,  CPU, 
and  network  intensive  pnoc esses  all  at  one  time  as  a  result  of  Lhe  nature  of  the  lab  or 
exercise.  Since  two  out  of  three  of  these  potential  boiLle necks  are  handled  by  the  CPU’s^ 
the  setup  purchased  by  the  department  should  contain  as  many  cores  as  possible,  while 
minimizing  the  number  of  actual  processors  to  cut  down  on  licensing  costs  [ESX  is 
licensed  by  processor,  independent  of  the  number  of  cores).  As  more  cores  become 
available,  more  physical  CPU's  that  can  be  assigned  to  a  virtual  machine  in  a  case  when 
there  are  less  images  than  [  cores  ]  +  [  processors  ].  and  the  more  load  sharing  that  can  be 
enabled  if  not, 

j.  Rccommmda  lions 

Based  on  this  study,  I  recommend  the  Department  continue  expanding  the  ESX 
infrastructure,  Because  of  the  overwhelming  stress  on  CPU's  during  intense  network 
operations,  which  will  be  crucial  to  Information  Assurance  and  Computer  Networking 
courses,  die  Department  should  purchase  servers  with  more  cores  per  socket  vice  overall 
number  of  cores.  Since  ESX  Licensing  costs  are  based  on  sockets,  nor  number  of  cores,  a 
contest  between  a  server  with  two  sockets  and  six  cores  per  processor,  and  four  sockets 
and  four  cores  per  processor  would  undoubtedly  gp  to  the  two  socket  server,  ESX 
licenses  cost  roughly  100  per  year  per  socket  as  of  the  time  of  this  study,  so  our 
current  setup  would  cost  us  roughly  $2200  per  year  to  fully  license. 

Another  recommendation  I  puL  forth  is  that  the  Department  purchase  more,  less- 
expensive  ESX  servers  vice  fewer  servers  with  rop-of-the-l  ine  specifications  Aside  from 
limiting  single  points  of  failure,  this  also  provides  opportunities  to  experiment  with  and 
study  technologies  such  as  V  Mm  or  ion,  in  which  running  images  can  be  migrared  from 
server  lo  server  in  real-time,  This  study  can  be  geared  toward  performance  or  the 
security  of  such  technologies,  both  of  which  are  growing  concerns  in  modern  Department 
of  Defense  and  production  environments. 

My  final  recommendation  is  broader  in  scope:  when  Lhe  servers  are  purchased, 
each  Information  Assurance  lab  should  be  outfitted  with  dedicated  servers,  and  the  two 
sandbox  networks  should  be  connected  to  allow  for  larger  scale  exercises  utilizing  the 
infrastructure  from  both  labs,  ESX  provides  an  excellent  framework  for  dynamic,  real¬ 
time  exercises  to  be  conducted,  which  is  nor  a  capability  of  the  pre-formulated  labs  we 


conduct  in  our  current  Jnformatlnn  Assurance  classes,  These  dynamic  exercises  will 
more  accurately  portray  die  situations  our  Information  Professionals,  Information 
Warfare  Officers,  and  graduates  in  aeneral  are  liltely  to  encounter  in  the  Fleet  and 
beyond. 


Appendix  A:  Uploading  and  tioiviilojclhig  images  liom  ait  ESXi  server 


/.  Download  using  vSphere  client 

a,  Connect  to  rhe  E5X  serve r  by  clicking  'vS  phene  Client/  entering  the  IP, 
username,  and  password 
li  Select  the  server  (  192.1 6S.  1.200) 

c.  Under  the  'summary1  tab,  look  on  the  right  side,  and  click  whichever  data- 
store  the  image  is  on: 


v 


ln  the  left  hand  column  of  the  'datastore  browser'  dialog,  click  on  die 
folder  of  ifie  image  you  wish  to  download,  and  select  tlie  download  button 


e,  Select  tine  destination  folder  you  wish  to  pul  tfie  image  into 

f.  Click  next,  and  download.  The  images  folder  will  be  available  in  the  di¬ 
rectory  you  specified. 
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r.  Hlr  ‘Neit1 * 3  and  complete  the  virtual  machine. 

15,  Browse  rhe  daiastore  you  created  the  machlm-  lit,  a*  detailed  In  pan  Ic 
Hu  Select  ihe  *F  directory'  your  VM  resides  in  on  the  left  hand  side  of  die  dia- 
kig ,  .ukL  select  upload 


I,  Select  ihe  vmdk  Ides  from  the  direciory  ol  Htr  inu“c-  con  are  in  tn%  to  up¬ 
load.  and  hit  'Next'  until  ii  upload s 

j.  In  die  main  vSphere  client  window,  riglit  elk  k  the  vmual  machine  you 
jusa  created,  and  select  Edit  Settings' 

It,.  Under  ihe  ■Hardware'  iab,  seSed  ■Add" 

3,  Select  'Hard  Disk',  then  'Use  an  existing  virtual  disk' 


m,  Browse  10  rhe  location  of  vmdk  file  in  Lhe  datasroie,  and  click  'Next'  until 
it  is  added. 

il  Bool  the  virtual  machine  by  right  clicking  it.  then  Power  ->  Power  On 

o,  View  it  by  right  clicking  on  it  again,  and  selecting  'Open  Console' 

Upload  using  VMware  Workstation 

a.  Open  VMware  Workstation,  and  select  ' Import/Export'  horn  the  Tile' 
menu  as  detailed  in  part  2 

b.  Select  'Virtual  Appliance'  from  the  combo  box 

c.  Select  File  System,  and  browse  to  where  the  image  is  stored  on  your  phys¬ 
ical  band  drive 

L  Note:  a  message  will  appear  here  stating  that  'The  dring?  specified 
is  li backup  image’- click  OK 

cL  Click  'Next'  until  you  get  to  specifying  the  destination  type. 

e.  Select  'VMware  Infrastructuie  Virtual  Machine,1  and  log  in  as  detailed,  in 
part  2c 

L  Select  the  datastoie  and  resource  pool  to  add  lhe  machine  to.  click  Next 

g.  Click  'Next'  through  any  error  messages 

Tl  Once  uploaded,  boot  your  machine  on  the  ESX  server  by  right  clicking  on 
it,  then  Power  ^  Power  On 

i.  View  it  by  riglir  clicking  on  iL  again,  and  selecting  'Open  Console' 


Appendix  ]J:  Dc H  PowcrEdge  2950  server  specinealioiis 


Model:  Del]  Poweredge  Energy  Smart  2950  ]]] 

Processor :  Quad-core  Intel  X  eon  L.  54 10  -  2x6  MB  Cacti e, 

1333  MHz  Front-side  Bus 

Memory;  SCB  Memory  GG7MHz  £4x2CB) 

Dual  Ranked  DIM  M's 


Storage:  Primary  Hard  Drive:  2.5'1,  73GB  JOK  RPM  Serial  Attached  SCSI 

-3 GBPS.  SATA 

Secondary  Hard  Drive:  450CB  2.5”  SATA,  10K  RPM, 


RAID:  Primary  Cont roller:  PERC6I  SAS  RAID  Controller,  2 \4 connectors,  256MB 
Cache 

Network;  Dual  Embedded  Broadcom  MetXLreme  ]J  570S,  Ci^bit  Ethernet 
CD- Drive:  DVD-RW 
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